Informing the Design of Deictic Behaviors of a Web Agent with Spoken and Sign Language Video Data

Deixis refers to linguistic features that relate utterances to the circumstances of space and time in which they occur. Pointing gestures are regarded as indicating an object, a location or a direction, which is discovered by projecting a straight line from the furthest point of the body part that has been extended outward, into the space that extends beyond the speaker [1]. Pointing show complex relations between multiple modalities such as gaze, hand, torso orientation and language [2]. The form of pointing adopted provides information about how the speaker wishes the object being indicated to be regarded [1]. Few virtual agents endowed with deictic capabilities [35] have been designed from experimental data collected with the target application. Furthermore, few studies have compared deixis in speaking and signing users [6]. In this paper we focus on the specification of deictic behaviors to be displayed by speaking and signing web agents during a presentation task. A video corpus was collected in the context of the target application in order to study the deictic behaviors in both vocal and signed languages. We also describe the DIVA web agents that we use for web based presentations.